class: title-slide, left, bottom # Combining a smooth information criterion with neural networks ---- ## **Andrew McInerney**, ** ** ### University of Limerick #### LMU, 07 July 2023 --- # Where is Limerick? -- .pull-left[ <img src="data:image/png;base64,#img/limerick-map.png" width="100%" style="display: block; margin: auto;" /> ] -- <img src="data:image/png;base64,#img/limerick-city.jpg" width="30%" style="display: block; margin: auto 0 auto auto;" /> <img src="data:image/png;base64,#img/king-johns.jpg" width="30%" style="display: block; margin: auto 0 auto auto;" /> --- # Limerick -- .pull-left[ <img src="data:image/png;base64,#img/hurling.jpg" width="70%" style="display: block; margin: auto;" /> <img src="data:image/png;base64,#img/colmcille.png" width="40%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="data:image/png;base64,#img/dolores.jpg" width="85%" style="display: block; margin: auto;" /> ] --- # Univeristy of Limerick -- .pull-left[ <img src="data:image/png;base64,#img/ul1.jpeg" width="100%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="data:image/png;base64,#img/ul2.jpg" width="100%" style="display: block; margin: auto;" /> ] --- # Background -- <img src="data:image/png;base64,#img/crt-logo.jpg" width="60%" style="display: block; margin: auto;" /> -- * Research: Neural networks from a statistical-modelling perspective -- <img src="data:image/png;base64,#img/packages.png" width="70%" style="display: block; margin: auto;" /> --- class: selectnn-slide # Model Selection <img src="data:image/png;base64,#img/modelsel.png" width="90%" style="display: block; margin: auto;" /> A Statistically-Based Approach to Feedforward Neural Network Model Selection (arXiv:2207.04248) --- class: selectnn-slide # Insurance: Model Selection ```r library(selectnn) nn <- selectnn(charges ~ ., data = insurance, Q = 8, n_init = 5) summary(nn) ``` -- ```{.bg-primary} ## [...] ## Number of input nodes: 4 ## Number of hidden nodes: 2 ## ## Value: 1218.738 ## Covariate Selected Delta.BIC ## smoker.yes Yes 2474.478 ## bmi Yes 919.500 ## age Yes 689.396 ## children Yes 13.702 ## [...] ``` --- class: interpretnn-slide # Interpreting FNNs Extend packages: **nnet**, **neuralnet**, **keras**, **torch** * Significance testing * Covariate-effect plots --- class: interpretnn-slide # Insurance: Model Summary ```r intnn <- interpretnn(nn) summary(intnn) ``` -- ```{.bg-primary} ## Coefficients: ## Weights | X^2 Pr(> X^2) ## age (-0.43***, 0.04) | 41.4363 1.01e-09 *** ## sex.male (0.08*, 0.13) | 5.5055 6.38e-02 . ## bmi (0.03, 2.19***) | 105.6106 0.00e+00 *** ## children (-0.08***, -0.11.) | 19.0146 7.43e-05 *** ## smoker.yes (-3.16***, -6.19***) | 250.6393 0.00e+00 *** ## region.northwest (0.07., 0.15) | 2.8437 2.41e-01 ## region.southeast (0.11*, 0.12) | 6.2560 4.38e-02 * ## region.southwest (0.15**, 0.14) | 10.8218 4.47e-03 ** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ``` --- class: interpretnn-slide # Insurance: Model Summary ```r plotnn(intnn) ``` -- <img src="data:image/png;base64,#img/insurance-plotnn.png" width="70%" style="display: block; margin: auto;" /> --- class: interpretnn-slide # Insurance: Covariate Effects ```r plot(intnn, conf_int = TRUE, which = c(1, 4)) ``` -- .pull-left[ <!-- --> ] -- .pull-right[ <!-- --> ] --- # Current Work -- <br> .pull-left[ <img src="data:image/png;base64,#img/kevin-meadhbh.png" width="100%" style="display: block; margin: auto;" /> ] -- .pull-right[ <img src="data:image/png;base64,#img/sic-publication.png" width="100%" style="display: block; margin: auto;" /> ] --- # Smooth Information Criterion $$ \text{IC} = -2\ell(\theta) + \lambda [\lVert \tilde\theta \rVert_0 + 1] $$ where `\(\lambda = \log(n)\)` for the BIC. -- Rearrange as an IC-based penalized likelihood: `$$\ell^{\text{IC}}(\theta) = \ell(\theta) - \frac{\log(n)}{2} [\lVert \tilde\theta \rVert_{0} + 1]$$` --- # Smooth Information Criterion Introduce "smooth `\(L_0\)` norm": `$$\lVert \theta \rVert_{0, \epsilon} = \sum_{j=1}^p \phi_\epsilon (\theta_j)$$` where $$ \phi_\epsilon(\theta_j) = \frac{{\theta_j^2}}{\theta_j^2 + \epsilon^2} $$ --- # Smooth Information Criterion <img src="data:image/png;base64,#img/smooth-l0.png" width="80%" style="display: block; margin: auto;" /> --- # Motivation -- * Tuning parameter automatically selected in one step <br> -- * Computationally advantageous --- # `\(\epsilon\)`-telescoping -- * Optimal `\(\epsilon\)` is zero -- * Smaller `\(\epsilon\)` `\(\implies\)` less numerically stable -- * Start with larger `\(\epsilon\)`, and "telescope" through a decreasing sequence of `\(\epsilon\)` values using warm starts --- # R Package <img src="data:image/png;base64,#img/smoothic.png" width="70%" style="display: block; margin: auto;" /> --- # Extending to Neural Networks `$$\mathbb{E}(y) = \text{NN}(X, \theta)$$` -- where `$$\text{NN}(X, \theta) = \phi_o \left[ \gamma_0+\sum_{k=1}^q \gamma_k \phi_h \left( \sum_{j=0}^p \omega_{jk}x_{j}\right) \right]$$` --- # Extending to Neural Networks We can then formulate a **smooth** BIC-based penalized likelihood: -- `\begin{equation*} \ell^{\text{SIC}}(\theta) = \ell(\theta) - \frac{\log(n)}{2} [\lVert \tilde\theta \rVert_{0, \epsilon} + q + 1], \end{equation*}` -- where `\begin{equation*} \ell(\theta)= -\frac{n}{2}\log(2\pi\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-\text{NN}(x_i))^2 \end{equation*}` --- # Extending to Group Sparsity -- The smooth approximation of the `\(L_0\)` norm can be written for groups as <!-- $$ --> <!-- \phi_\epsilon(\theta^{(g)}) = \lvert \theta^{ (g) } \rvert \frac{ {\lVert \theta^{ (g) } \rVert}_2^2}{ {\lVert \theta^{ (g) } \rVert}_2^2 + \epsilon^2}. --> <!-- $$ --> $$ \text{card}(\theta) \times \phi_\epsilon(||\theta||_2^2) = \text{card}(\theta) \times \frac{||\theta||_2^2}{||\theta||_2^2 + \epsilon^2}. $$ --- class: inputgroup-slide # Group Sparisty ## Input-neuron penalization <!-- \begin{equation*} --> <!-- \ell^{\text{IN-SIC}}(\theta) = \ell(\theta) - \frac{\log(n)}{2} \left[\sum_{j=1}^{p} \lVert \omega_{j} \rVert_{0, \epsilon} + \lVert \tilde\gamma \rVert_{0, \epsilon} + q + 1\right], --> <!-- \end{equation*} --> <font size="6"> `\begin{equation*} \ell^{\text{IN-SIC}}(\theta) = \ell(\theta) - \frac{\log(n)}{2} \left[q \times \sum_{j=1}^p\big\Vert\Vert\omega_j\Vert_2^2\big\Vert_{0,\epsilon} + \lVert \tilde\gamma \rVert_{0, \epsilon} + q + 1\right] \end{equation*}` </font> where `\(\omega_{j} = (\omega_{j1},\omega_{j2},\dotsc,\omega_{jq})^T\)` --- class: hiddengroup-slide # Group Sparisty ## Hidden-neuron penalization <!-- \begin{equation*} --> <!-- \ell^{\text{HN-SIC}}(\theta) = \ell(\theta) - \frac{\log(n)}{2} \left[\sum_{k=1}^{q} \lVert \theta^{(k)} \rVert_{0, \epsilon} + q + 1\right], --> <!-- \end{equation*} --> <font size="6"> `\begin{equation*} \ell^{\text{HN-SIC}}(\theta) = \ell(\theta) - \frac{\log(n)}{2} \left[(p+1) \times \sum_{k=1}^q\big\Vert\Vert\theta^{(k)}\Vert_2^2\big\Vert_{0,\epsilon} + q + 1\right] \end{equation*}` </font> where `\(\theta^{(k)} = (\omega_{1k},\omega_{2k},\dotsc,\omega_{pk}, \gamma_k)^T\)` --- # Combined Penalty * Implement a group penalty and the single-parameter penalty in one optimization procedure * Start with group penalization and telescope through the `\(\epsilon\)` values until some predefined value, `\(\tau\)` * Switch to single-parameter penalization for the remainder of the `\(\epsilon\)` values --- # Approaches * Single-parameter penalization * Input-neuron penalization * Hidden-neuron penalization * Combined approaches (perform group penalization initially and then switch to single-parameter penalization) --- # Preliminary Simulation <img src="data:image/png;base64,#img/nn-sim-plot.png" width="100%" style="display: block; margin: auto;" /> --- # Preliminary Results <img src="data:image/png;base64,#img/prelim-results.png" width="90%" style="display: block; margin: auto;" /> -- <img src="data:image/png;base64,#img/prelim-results-extra.png" width="90%" style="display: block; margin: auto;" /> --- class: bigger # References * <font size="5">McInerney, A., & Burke, K. (2022). A statistically-based approach to feedforward neural network model selection. <i>arXiv preprint arXiv:2207.04248</i>. </font> * <font size="5">McInerney, A., & Burke, K. (2023). Interpreting feedforward neural networks as statistical models. <i>To appear on arXiv</i>. </font> * <font size="5">O’Neill, M. and Burke, K. (2023). Variable selection using a smooth information criterion for distributional regression models. <i>Statistics and Computing, 33(3), p.71</i>. </font> ### R Packages ```r devtools::install_github(c("andrew-mcinerney/selectnn", "andrew-mcinerney/interpretnn")) ```
<font size="5.5">andrew-mcinerney</font>
<font size="5.5">@amcinerney_</font>
<font size="5.5">andrew.mcinerney@ul.ie</font>